Using generative AI to investigate medical imagery models and datasets.

BACKGROUND: AI models have shown promise in performing many medical imaging tasks. However, our ability to explain what signals these models have learned is severely lacking. Explanations are needed in order to increase the trust of doctors in AI-based models, especially in domains where AI prediction capabilities surpass those of humans. Moreover, such explanations could enable novel scientific discovery by uncovering signals in the data that aren't yet known to experts. METHODS: In this paper, we present a workflow for generating hypotheses to understand which visual signals in images are correlated with a classification model's predictions for a given task. This approach leverages an automatic visual explanation algorithm followed by interdisciplinary expert review. We propose the following 4 steps: (i) Train a classifier to perform a given task to assess whether the imagery indeed contains signals relevant to the task; (ii) Train a StyleGAN-based image generator with an architecture that enables guidance by the classifier ("StylEx"); (iii) Automatically detect, extract, and visualize the top visual attributes that the classifier is sensitive towards. For visualization, we independently modify each of these attributes to generate counterfactual visualizations for a set of images (i.e., what the image would look like with the attribute increased or decreased); (iv) Formulate hypotheses for the underlying mechanisms, to stimulate future research. Specifically, present the discovered attributes and corresponding counterfactual visualizations to an interdisciplinary panel of experts so that hypotheses can account for social and structural determinants of health (e.g., whether the attributes correspond to known patho-physiological or socio-cultural phenomena, or could be novel discoveries). FINDINGS: To demonstrate the broad applicability of our approach, we present results on eight prediction tasks across three medical imaging modalities-retinal fundus photographs, external eye photographs, and chest radiographs. We showcase examples where many of the automatically-learned attributes clearly capture clinically known features (e.g., types of cataract, enlarged heart), and demonstrate automatically-learned confounders that arise from factors beyond physiological mechanisms (e.g., chest X-ray underexposure is correlated with the classifier predicting abnormality, and eye makeup is correlated with the classifier predicting low hemoglobin levels). We further show that our method reveals a number of physiologically plausible, previously-unknown attributes based on the literature (e.g., differences in the fundus associated with self-reported sex, which were previously unknown). INTERPRETATION: Our approach enables hypotheses generation via attribute visualizations and has the potential to enable researchers to better understand, improve their assessment, and extract new knowledge from AI-based models, as well as debug and design better datasets. Though not designed to infer causality, importantly, we highlight that attributes generated by our framework can capture phenomena beyond physiology or pathophysiology, reflecting the real world nature of healthcare delivery and socio-cultural factors, and hence interdisciplinary perspectives are critical in these investigations. Finally, we will release code to help researchers train their own StylEx models and analyze their predictive tasks of interest, and use the methodology presented in this paper for responsible interpretation of the revealed attributes. FUNDING: Google.

Risk Stratification for Diabetic Retinopathy Screening Order Using Deep Learning: A Multicenter Prospective Study.

Purpose: Real-world evaluation of a deep learning model that prioritizes patients based on risk of progression to moderate or worse (MOD+) diabetic retinopathy (DR). Methods: This nonrandomized, single-arm, prospective, interventional study included patients attending DR screening at four centers across Thailand from September 2019 to January 2020, with mild or no DR. Fundus photographs were input into the model, and patients were scheduled for their subsequent screening from September 2020 to January 2021 in order of predicted risk. Evaluation focused on model sensitivity, defined as correctly ranking patients that developed MOD+ within the first 50% of subsequent screens. Results: We analyzed 1,757 patients, of which 52 (3.0%) developed MOD+. Using the model-proposed order, the model's sensitivity was 90.4%. Both the model-proposed order and mild/no DR plus HbA1c had significantly higher sensitivity than the random order (P < 0.001). Excluding one major (rural) site that had practical implementation challenges, the remaining sites included 567 patients and 15 (2.6%) developed MOD+. Here, the model-proposed order achieved 86.7% versus 73.3% for the ranking that used DR grade and hemoglobin A1c. Conclusions: The model can help prioritize follow-up visits for the largest subgroups of DR patients (those with no or mild DR). Further research is needed to evaluate the impact on clinical management and outcomes. Translational Relevance: Deep learning demonstrated potential for risk stratification in DR screening. However, real-world practicalities must be resolved to fully realize the benefit.

A deep learning model for novel systemic biomarkers in photographs of the external eye: a retrospective study.

BACKGROUND: Photographs of the external eye were recently shown to reveal signs of diabetic retinal disease and elevated glycated haemoglobin. This study aimed to test the hypothesis that external eye photographs contain information about additional systemic medical conditions. METHODS: We developed a deep learning system (DLS) that takes external eye photographs as input and predicts systemic parameters, such as those related to the liver (albumin, aspartate aminotransferase [AST]); kidney (estimated glomerular filtration rate [eGFR], urine albumin-to-creatinine ratio [ACR]); bone or mineral (calcium); thyroid (thyroid stimulating hormone); and blood (haemoglobin, white blood cells [WBC], platelets). This DLS was trained using 123 130 images from 38 398 patients with diabetes undergoing diabetic eye screening in 11 sites across Los Angeles county, CA, USA. Evaluation focused on nine prespecified systemic parameters and leveraged three validation sets (A, B, C) spanning 25 510 patients with and without diabetes undergoing eye screening in three independent sites in Los Angeles county, CA, and the greater Atlanta area, GA, USA. We compared performance against baseline models incorporating available clinicodemographic variables (eg, age, sex, race and ethnicity, years with diabetes). FINDINGS: Relative to the baseline, the DLS achieved statistically significant superior performance at detecting AST >36·0 U/L, calcium <8·6 mg/dL, eGFR <60·0 mL/min/1·73 m2, haemoglobin <11·0 g/dL, platelets <150·0 × 103/µL, ACR ≥300 mg/g, and WBC <4·0 × 103/µL on validation set A (a population resembling the development datasets), with the area under the receiver operating characteristic curve (AUC) of the DLS exceeding that of the baseline by 5·3-19·9% (absolute differences in AUC). On validation sets B and C, with substantial patient population differences compared with the development datasets, the DLS outperformed the baseline for ACR ≥300·0 mg/g and haemoglobin <11·0 g/dL by 7·3-13·2%. INTERPRETATION: We found further evidence that external eye photographs contain biomarkers spanning multiple organ systems. Such biomarkers could enable accessible and non-invasive screening of disease. Further work is needed to understand the translational implications. FUNDING: Google.

Detection of signs of disease in external photographs of the eyes via deep learning.

Retinal fundus photographs can be used to detect a range of retinal conditions. Here we show that deep-learning models trained instead on external photographs of the eyes can be used to detect diabetic retinopathy (DR), diabetic macular oedema and poor blood glucose control. We developed the models using eye photographs from 145,832 patients with diabetes from 301 DR screening sites and evaluated the models on four tasks and four validation datasets with a total of 48,644 patients from 198 additional screening sites. For all four tasks, the predictive performance of the deep-learning models was significantly higher than the performance of logistic regression models using self-reported demographic and medical history data, and the predictions generalized to patients with dilated pupils, to patients from a different DR screening programme and to a general eye care programme that included diabetics and non-diabetics. We also explored the use of the deep-learning models for the detection of elevated lipid levels. The utility of external eye photographs for the diagnosis and management of diseases should be further validated with images from different cameras and patient populations.

Pembrolizumab and epacadostat induced fatal myocarditis and myositis presenting as a case of ptosis and ophthalmoplegia.

We report the first case of fatal myocarditis presenting as bilateral ptosis in a patient on combination therapy with pembrolizumab and epacadostat. An 83 year-old man with stage III high-grade urothelial carcinoma presented with acute onset droopy eyelids one month after starting pembrolizumab and epacadostat. Exam showed myogenic ptosis and ophthalmoplegia. He was later found to have acute myocarditis with complete heart block and subsequently passed away. Pembrolizumab in combination with epacadostat can induce a potentially fatal myocarditis. Although immune mediated myocarditis is a rare established side effect, more reported fatalities are needed in the literature to highlight the urgency for standardized cardiac monitoring of even asymptomatic patients to prevent fatal outcomes, as well as a consensus on treatment protocols. Cancer immunotherapy complications are not well known to ophthalmologists. This case is unique in that the presenting sign was ptosis, which prompted the patient to call his ophthalmologist first.

Prevalence and nature of early glaucomatous defects in the central 10° of the visual field.

IMPORTANCE: The macula is essential for visual functioning and is known to be affected even in early glaucoma. However, little is currently understood about the prevalence and nature of central vision loss in early glaucoma. OBJECTIVE: To determine the prevalence and characteristics of visual field (VF) defects in the central 10° in glaucoma suspects and patients with mild glaucoma using a prospective design. DESIGN, SETTING, AND PARTICIPANTS: This prospective observational cohort study was conducted at an outpatient glaucoma specialty clinic. One hundred eyes from 74 patients with glaucomatous optic neuropathy and a 24-2 VF with mean deviation better than -6 dB were prospectively studied and tested with a 10-2 test. MAIN OUTCOMES AND MEASURES: Reliable: VF hemifields were classified as abnormal based on a cluster criterion, and abnormal 10-2 VFs were categorized based on the pattern of abnormal points: arcuatelike, widespread, or other. In addition, at each point of the 10-2 VF, the total deviation values were averaged across eyes and the number of abnormal points with total deviation values below a specific criterion level were calculated. RESULTS There appeared to be as many abnormal 10-2 hemifields (53%) as abnormal 24-2 hemifields (59%). Of the eyes with normal 24-2 hemifields, 16% were classified as abnormal when the 10-2 test was used. Of the abnormal 10-2 hemifields, 68%, 8%, and 25% were arcuatelike, widespread, and other, respectively. The average total deviation values and number of abnormal points plots revealed superior VF defects that were deeper and closer to fixation than those in the inferior VF. CONCLUSIONS AND RELEVANCE: The 10-2 VF was abnormal in nearly as many hemifields as was the 24-2 VF, including some with normal 24-2 VF, suggesting that the 24-2 test is not optimal for detecting early damage of the macula. The pattern of the defects was in agreement with a recent model of macular damage.

Infiltration of the optic chiasm, nerve, and disc by gliomatosis cerebri.

An 18-year-old man with gliomatosis cerebri (GC) developed tumor infiltration of the optic chiasm and right optic nerve including the optic disc. Although papilledema often is seen with GC, tumor invasion of the optic nerve head is observed.

Analysis of the alternative splicing of an FGFR2 transcript due to a novel 5' splice site mutation (1084+1G>A): case report.

Craniosynostosis is characterized by premature fusion of one or more cranial sutures and is associated with mutations in fibroblast growth factor receptor (FGFR) genes. Here we describe a novel mutation (1084+1G>A) in the FGFR2 gene of a patient with isolated bicoronal synostosis. We detected two isoforms that result from the mutation and are characterized, respectively, by exon skipping and the use of a cryptic splice site. Interestingly, the alternatively spliced forms of FGFR2 appear to induce fusion of the cranial sutures suggesting that the mutation acts via a gain-of-function mechanism rather than a loss of protein functionality.

Mitochondrial DNA analysis by multiplex denaturing high-performance liquid chromatography and selective sequencing in pediatric patients with cardiomyopathy.

PURPOSE: Mitochondrial DNA testing is typically performed by targeted mutation analysis only. We applied a more comprehensive approach to study the mitochondrial genome in 24 pediatric patients with idiopathic cardiomyopathy. METHODS: Patients in the cohort did not show overt multisystemic disease and were previously tested for mutations in a subset of structural genes associated with cardiomyopathy. Mutation screening of the mitochondrial DNA by multiplex denaturing high-performance liquid chromatography was complemented by sequence analysis. RESULTS: We identified 130 individual (unique) sequence changes. Among several potentially pathogenic changes, a novel heteroplasmic mutation in nicotinamide adenine dinucleotide dehydrogenase subunit 4 (10677G>A) was identified in one fraternal twin with worse clinical symptoms than his sibling. Another proband carried homoplasmic mutation 13708G>A (in nicotinamide adenine dinucleotide dehydrogenase subunit 5) that has been associated with Leber's hereditary optic neuropathy. CONCLUSIONS: Changes in mitochondrial DNA may represent a relatively rare cause of idiopathic pediatric cardiomyopathies and/or influence their phenotypic expression. Interpretation of variants with uncertain pathogenicity, however, currently impedes clinical diagnostic use of comprehensive mitochondrial DNA testing. Whereas combined use of multiplex denaturing high-performance liquid chromatography and sequencing is more comprehensive than targeted mutation analysis, measurement of additional functional parameters, such as tissue respiratory chain activity, remains important to establishing a definitive diagnosis.